Skip to content

feat: CRD value accessors (native modules + raw objects, src + chart)#99

Merged
arnarg merged 10 commits into
arnarg:mainfrom
sini:feat/crd-native-module
Jun 8, 2026
Merged

feat: CRD value accessors (native modules + raw objects, src + chart)#99
arnarg merged 10 commits into
arnarg:mainfrom
sini:feat/crd-native-module

Conversation

@sini

@sini sini commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Adds CRD generators that return Nix values instead of building a generated
.nix file, plus their chart and raw-object counterparts — completing the API
matrix.

Rebased onto main; #98 (kindFilter on fromCRD) merged as squashed
b575aac, so its commit no longer appears in this PR's range.

Summary

src-based chart-based
types → file fromCRD fromChartCRD
types → module fromCRDModule fromChartCRDModule
raw objects crdObjects crdObjectsFromChart

Motivation

fromCRD renders the resource options to Nix source, nixfmts it, writes a
.nix file, and you import it back — a serialize→deserialize round-trip plus
an import-from-derivation. For flake/programmatic consumers that just want the
options, that file is pure overhead. nixidy.applicationImports already accepts
functionTo attrs, so fromCRDModule returns the module value directly and
drops straight in — no file, no import IFD.

crdObjects / crdObjectsFromChart expose the raw CustomResourceDefinition
manifests as values (e.g. to apply them to a cluster), deliberately
deployment-agnostic — they hand back objects and let the caller decide.

What's in here

  1. fromCRDModule — native module value; no generated file.
  2. Backend unification — the schema walk is extracted into one backend-parameterized walk.nix, driving a text backend (source, for the committed-file workflow) and a value backend (live module). generator.nix/module.nix become thin assemblers over the one walk, so the branch logic lives in a single place.
  3. crdObjects — raw CRD manifests from a src.
  4. Chart accessors fromChartCRDModule / crdObjectsFromChart, sharing one helm template (via klib.buildHelmChart); kubeVersion is now overridable across the chart family (incl. fromChartCRD), defaulting to nixidy's nixpkgs version — previously hardcoded.
  5. Parameter rename — the overloaded crds argument is split into crdFiles (source-based) and kindFilter (chart-based), since it meant a list of YAML files in one family and a kind filter in the other. crds is kept as a deprecated alias on all six accessors (new arg wins; falling back warns and points at the new name).
  6. Documentation — typed_resources.md (matrix-led, with the new accessors + kubeVersion), architecture.md (shared walk + backends), llms.txt.
  7. Tests (nix run .#crdAccessorTest).

Backwards compatibility / verification

  • fromCRD / fromChartCRD file output is unchanged — verified nix build .#generators.argocd yields the same store path before/after the backend refactor. fromChartCRD only gains an optional kubeVersion with its prior default, plus the crdskindFilter alias.
  • Green: moduleTests 42/42, crd2jsonschemaTest 18/18, crdAccessorTest (10 checks: chart↔src equivalence, kubeVersion honored, deprecated-crds-alias resolution, specialMapKeys on the value backend), staticCheck.

Done

Follow-ups (separate PR)

  • Demo template — a flake-native example using fromCRDModule via applicationImports (and crdObjects to deploy the CRDs). Deferred to its own PR: templates are inherently opinionated and warrant separate review.

Notes for review

  • Inherent duplication worth a second opinion: the per-generated-module runtime
    helpers exist both as source (inlined by the text backend, so committed
    standalone files stay self-contained) and as values (runtime.nix, for the
    native modules). Live types/functions can't be serialized back to source, and
    committed files can't reference nixidy internals — so the two forms must
    coexist. The schema walk (the real divergence risk) is single-sourced.

@sini sini changed the title feat(generators): CRD value accessors (native modules + raw objects, src + chart) feat: CRD value accessors (native modules + raw objects, src + chart) Jun 4, 2026
@sini

sini commented Jun 4, 2026

Copy link
Copy Markdown
Contributor Author

Using this here to automatically inject CRDs into evaluation for type checking/defaults, without maintaining static CRD nix files.

@arnarg

arnarg commented Jun 7, 2026

Copy link
Copy Markdown
Owner

I have considered adding resource options generation at evaluation time like this before but I was concerned about evaluation speed with such a setup. Have you noticed major slowdowns with this setup? I'm not opposed to this setup in general but I will need to some time to review your changes.

Thanks for the effort in improving nixidy! :)

@arnarg

arnarg commented Jun 7, 2026

Copy link
Copy Markdown
Owner

Also, I've gotta ask, was any of this LLM generated? I don't have any policy that disallows LLM usage, it's just good to know when reviewing it. Also there's no need for a assisted-by header or anything, I'm just curious to know :)

@sini

sini commented Jun 7, 2026

Copy link
Copy Markdown
Contributor Author

Yes, I had Opus 4.8 extract the pattern out of my config for upstreaming/sharing. Feel free to critique it mercilessly. :)

Comment thread pkgs/generators/default.nix Outdated
@arnarg

arnarg commented Jun 8, 2026

Copy link
Copy Markdown
Owner

Also, I'd like to take this opportunity to unify the parameter naming between the fromCRD and fromChartCRD (and your new variants of the two). The biggest issue is that they both have a parameter called crds but they have different meanings. I'd like to change it to the following:

  • fromCRD (and your new variants) rename the crds parameter to crdFiles.
  • fromChartCRD (and your new variants) rename the crds parameter to kindFilter to match the newly added parameter, as they have the same functionality.

Ideally, they should both still take a crds parameter that has a lower priority than the new parameter, for backwards compatibility.

What do you think about that?

@sini

sini commented Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

Finally have some time to properly sit down with a computer and give richer feedback.

Have you noticed major slowdowns with this setup?

No; in practice these stay cached within the nix store and make it much easier for me to bump chart versions -- it's no longer a multi-commit juggling act.

Adjacently I did try folding the IFDs on nixidy/kubenix down to just one call; but it actually wasn't any faster for me at the end of the day. That's why the version here is the re-expanded one. :) I've been running a variant of this pattern for ~3-4 months, but my folded-IFD version not being worth contributing is what delayed my upstreaming.

I also explored conceptually minimizing IFD as much as possible / moving some eval to pure nix as an academic exercise; but without a reliable native/C-based builtins.fromYAML, I found it's not worth it. IFD is just the nature of the beast for this problem today.

Side note; it's actually been nice to programmatically manage these this way; in my personal argocd deployment I have a 'bootstrap' app that includes all global CRD resources + namespaces so there aren't any cross-app namespace race conditions on things like cilium firewall rules/etc.

[...rename crds...] What do you think about that?

I think it's a good idea; will apply it to all 6 interfaces. I'll add a deprecation warning with rename instructions so crds can be removed in the future / avoid incurring legacy tech debt forever instead of just handling the backwards compatibility indefinitely.

edit:

for clarity; when I said 'manage all crd resources' I meant it in the context of 'atm machine' :D -- declared resources are still in the their apps; the custom resource definitions themselves are the only thing cloned.

sini added 7 commits June 8, 2026 08:56
…ound-trip

`fromCRD` generates Nix *source*, writes a `.nix` file, and you `import` it
back — a serialize/deserialize round-trip plus an import-from-derivation. For
flake consumers that just want the resource options, that intermediate file
is pure overhead.

`fromCRDModule` returns the resource definitions as a module *value*
(`{ lib, options, config, ... }: { ... }`) instead. It drops straight into
`nixidy.applicationImports` (which already accepts `functionTo attrs`), so
there is no generated file and no import IFD — only the unavoidable Python
crd2jsonschema parse, now shared between both backends via the extracted
`crdSchema` helper.

- runtime.nix: the per-module boilerplate that generator.nix inlines as
  source (submoduleOf, coerce-attrs-to-list-by-key, the str/coercedTo type
  overrides, defaults handling, ...), extracted as real functions.
- module.nix: a value-emitting port of generator.nix's schema walk
  (genDefinitions / mapType / genResourceOptions).
- equiv-test.nix + `nix run .#crdModuleTest`: asserts fromCRDModule renders
  identically to fromCRD for a CRD that exercises submodules, list-by-key
  coercion and the global ObjectMeta metadata ref.

The text generator (`fromCRD`) is unchanged — verified to produce
byte-identical output — and remains for the CLI "commit generated types"
workflow, which needs source files.
The fromCRDModule prototype duplicated generator.nix's schema walk
(genDefinitions / mapType / genResourceOptions) inside module.nix — two
hand-kept twins of the type/coercion branch logic, the part most likely to
drift (flagged in review).

Extract the walk into walk.nix, parameterized by a `backend` of emit
combinators (types, mkOption, submoduleOf, coerce-attrs-to-list-by-key, ...).
Two backends implement it:
  - backend-text.nix  -> Nix source fragments; generator.nix assembles them
                         into the committable standalone .nix file
  - backend-value.nix -> live values via runtime.nix; module.nix assembles
                         them into a module value

generator.nix and module.nix are now thin assemblers over the one walk; the
branch logic exists in a single place (net -197 lines).

Verified: file generator output byte-identical (argocd store hash unchanged);
fromCRDModule still renders identically to fromCRD (nix run .#crdModuleTest);
moduleTests 42/42; crd2jsonschema 18/18; staticCheck clean.

Remaining (irreducible) duplication: the runtime helper block exists as live
functions in runtime.nix and as inlined source in generator.nix's template.
Committed standalone files can't reference nixidy internals, and live
types/functions can't be serialized back to source — so the two forms must
coexist. The walk (the real divergence risk) is no longer duplicated.
crdObjects { src, crds, kindFilter ? [ ] } returns the raw
CustomResourceDefinition objects from a set of CRD YAML files as values —
the objects counterpart to fromCRD's types. Deployment-agnostic: it hands
back the manifests and lets the caller decide what to do with them (e.g.
apply them to a cluster), rather than baking in an install mechanism.

Reuses fromChartCRD's isWanted filter shape; kindFilter mirrors fromCRD's
(empty = every CRD).
Add the chart counterparts of the src-based accessors:
  - fromChartCRDModule  → resource type module from a chart's CRDs
  - crdObjectsFromChart → raw CustomResourceDefinition manifests from a chart

Both share an internal mkChartCRDsYaml (raw `helm template --include-crds`),
so calling both with identical args templates the chart once. The output is
raw helm YAML parsed directly downstream — no klib.fromHelm/yq-go
re-serialization, avoiding its number-coercion quirks.

kubeVersion is now a first-class, overridable argument across the chart
family (fromChartCRD too), defaulting to nixidy's nixpkgs version. Consumers
templating CRDs for a cluster on a different Kubernetes version can target it
explicitly — previously the version was hardcoded to nixidy's pkgs.

Completes the CRD API matrix: {fromCRD, fromCRDModule, crdObjects} × {src,
chart}. fromChartCRD is unchanged for existing callers (same default).
Expand the accessor test (renamed crdModuleTest -> crdAccessorTest) from
fromCRD<->fromCRDModule equivalence to the full value-accessor surface:

- fromChartCRDModule renders identically to fromCRDModule for the same CRD
  (a local helm chart shipping the CRD in crds/, copied verbatim).
- crdObjects returns the raw CRD manifests; kindFilter narrows (hit + miss).
- crdObjectsFromChart returns the same objects as crdObjects.
- the chart accessors honor kubeVersion: a templated CRD embedding
  .Capabilities.KubeVersion renders v1.31 vs v1.40 observably differently.

Seven checks; the derivation builds iff all pass.
Drop the hand-rolled helm-template derivation in favor of
klib.buildHelmChart, per review. Materialize its single-file output as
$out/crds.yaml via runCommand cp rather than linkFarm: crdObjects reads
crds.yaml at eval time, and a linkFarm symlink would make that read
follow into buildHelmChart's separate output (a store path absent from
the readFile string's context) which pure eval forbids. A real file
keeps the read in-context.
`crds` meant two different things: a list of CRD YAML files (src-based
fromCRD/fromCRDModule/crdObjects) and a kind filter (chart-based
fromChartCRD/fromChartCRDModule/crdObjectsFromChart). Split into two
self-describing names:

  - src-based:   crds -> crdFiles
  - chart-based: crds -> kindFilter  (same role as fromCRD's kindFilter)

Every function keeps `crds` as a deprecated alias (new arg wins; falling
back to `crds` emits a lib.warn pointing at the replacement) via a shared
`renamedArg` helper, so existing callers keep working. The crd2jsonschema
JSON contract is unchanged (still keyed `crds`).

Tests cover both new names and the deprecated alias (9/9).
@sini
sini force-pushed the feat/crd-native-module branch from 2e35200 to a5e6a28 Compare June 8, 2026 15:57
sini added 2 commits June 8, 2026 09:11
…s clash

Address review feedback:

- renamedArg now warns (lib.warnIf) when both the deprecated `crds` and the
  new arg are supplied, instead of silently ignoring `crds`.
- Add a crdAccessorTest check exercising the value backend's specialMapKeys
  branch (module-only arg, no file-backend counterpart to diff against): two
  name-less ports with distinct merge-key values stay separate; the empty
  default collapses them.
…r rename

- typed_resources.md: lead the CRD section with the output-shape × source
  matrix; add subsections for fromCRDModule/fromChartCRDModule (module value,
  no generated file) and crdObjects/crdObjectsFromChart (raw manifests);
  document the kubeVersion chart argument; rename crds → crdFiles (src) /
  kindFilter (chart) in all examples with a deprecation note.
- architecture.md: describe the shared walk + text/value backends and the
  six-accessor matrix; update the CRD example args.
- llms.txt: mention the module-value and raw-object accessors.
@sini
sini marked this pull request as ready for review June 8, 2026 16:38
@sini

sini commented Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

Updated the PR description. I'm going to defer the proposed example/demo template for followup, as that will likely have it's own set of review feedback and want to decouple the concerns.

@sini
sini requested a review from arnarg June 8, 2026 16:41

@arnarg arnarg left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have tested this locally and read through the code and I think this looks great. Love the de-duplication with the different backends.

Amazing work! ⭐

@arnarg
arnarg merged commit a50655b into arnarg:main Jun 8, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants